Massive Memory Leak

Got a massive DXL script dealing with massive data that causes DOORS to run out of memory ~2gb when dealing with a massive module. DOORS retains its memory even after the offending dialog is closed even when all modules are closed.

Which of these actions would tend to cause memory leak associated with DOORS and not associated with the Dialog/Script doing the actions?

o string/bool extraction from an 'Array'.
o looping through a 'Skip' list getting the (string key skp) values.
o writing a line to a log file, closing the 'Stream', then re-opening the 'Stream'.
o String concatenation
o Retrieving a string from a Buffer.

This is an old ugly program I inherited and massively enhanced but its very complicated and sloppily written. Not sure where else to look.

  • Louie

llandale - Mon Jun 20 12:12:18 EDT 2011

Re: Massive Memory Leak
Mathias Mamsch - Mon Jun 20 13:14:46 EDT 2011

There is a couple of possibilities for a leak. Usually I encountered memory leaks with either a bug (had one with David a little while ago), or the memory of strings buffers or OLE objects is leaked. It is pretty hard to fill up 2 GB of memory by creating (and not deleting) Skip lists. A program like this would not finish, since it will become slower and slower this way, before memory would even reach critical values. You can also forget about bool, ints, etc. they are always stored on the stack, so this would give you a stack overflow.

o string/bool extraction from an 'Array'.
o looping through a 'Skip' list getting the (string key skp) values.

Nope. Looping through arrays, skips, etc. does not create new objects it will only give you references to already existing objects.

o writing a line to a log file, closing the 'Stream', then re-opening the 'Stream'.
Maybe. There could be the possibility that the Stream objects are leaked. But I guess this would be a pretty easy one to check by just commenting out the function.

o String concatenation
Oh yes. You can create a lot of string memory by unnecessary string handling. But I would look for large strings first, especially parsing richtext attribute values. Because generating 2GB from string concatenations is also pretty hard.

o Retrieving a string from a Buffer.
Oh yes. Same thing as string concatenation. This is the most common memory leak. Because there are two perms
Buffer ::[] (Buffer, Range_) and string ::[] (Buffer, Range_) there is a big chance that your code will use the buffer variant thereby leaking the intermediate buffer. So stuff like this always leaks memory:
  

Buffer buf = create(); buf = "123456789"
 
string s = "Part: " buf[2:5] "\n"

 


In this code buf[2:5] will always call the buffer variant to create a temporary buffer, then concatenate it to the temporary string, then leak the temporary buffer.

If you do not get results from manually tracking down the leak I have some library code that will help tracking the memory leak down. What DOORS version are you on?

Hope that helps, Regards, Mathias

 

 

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

 

 

Re: Massive Memory Leak
llandale - Mon Jun 20 14:03:09 EDT 2011

Mathias Mamsch - Mon Jun 20 13:14:46 EDT 2011

There is a couple of possibilities for a leak. Usually I encountered memory leaks with either a bug (had one with David a little while ago), or the memory of strings buffers or OLE objects is leaked. It is pretty hard to fill up 2 GB of memory by creating (and not deleting) Skip lists. A program like this would not finish, since it will become slower and slower this way, before memory would even reach critical values. You can also forget about bool, ints, etc. they are always stored on the stack, so this would give you a stack overflow.

o string/bool extraction from an 'Array'.
o looping through a 'Skip' list getting the (string key skp) values.

Nope. Looping through arrays, skips, etc. does not create new objects it will only give you references to already existing objects.

o writing a line to a log file, closing the 'Stream', then re-opening the 'Stream'.
Maybe. There could be the possibility that the Stream objects are leaked. But I guess this would be a pretty easy one to check by just commenting out the function.

o String concatenation
Oh yes. You can create a lot of string memory by unnecessary string handling. But I would look for large strings first, especially parsing richtext attribute values. Because generating 2GB from string concatenations is also pretty hard.

o Retrieving a string from a Buffer.
Oh yes. Same thing as string concatenation. This is the most common memory leak. Because there are two perms
Buffer ::[] (Buffer, Range_) and string ::[] (Buffer, Range_) there is a big chance that your code will use the buffer variant thereby leaking the intermediate buffer. So stuff like this always leaks memory:
  

Buffer buf = create(); buf = "123456789"
 
string s = "Part: " buf[2:5] "\n"

 


In this code buf[2:5] will always call the buffer variant to create a temporary buffer, then concatenate it to the temporary string, then leak the temporary buffer.

If you do not get results from manually tracking down the leak I have some library code that will help tracking the memory leak down. What DOORS version are you on?

Hope that helps, Regards, Mathias

 

 

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

 

 

The fundamental awkwardness of this situation is that its 8 year old code that my previous buddies are running in a classified environment for a new Project to which I have no access. The code is particular sloppy, far below my current standards.

Anyway, you got some notion on the difference between memory leak that is indeed cleaned up with the DXL/Dialog ends, and leak that remains residual?

As for your buffer extraction, I thought it odd about those intermediate returned Buffers, will be more mindful about those functions. I suppose I should work on a function that extracts from a buffer without any leak:

string fExtract(Buffer buf, int iFrom, iTo)
{ return(string bufiFrom, iTo) //??
}

bufResults += NameMod
bufResults += "\t"
bufResults += fExtract(bufInfo, 3, 22)
bufResults += "\n"

These do not produce leaks do they?
Buffer ::+=(Buffer,char) BufferCatChar
Buffer ::+=(Buffer,Buffer) BufferCatBuf
Buffer ::+=(Buffer,string) BufferCat

  • Louie

Re: Massive Memory Leak
Mathias Mamsch - Mon Jun 20 15:47:34 EDT 2011

llandale - Mon Jun 20 14:03:09 EDT 2011
The fundamental awkwardness of this situation is that its 8 year old code that my previous buddies are running in a classified environment for a new Project to which I have no access. The code is particular sloppy, far below my current standards.

Anyway, you got some notion on the difference between memory leak that is indeed cleaned up with the DXL/Dialog ends, and leak that remains residual?

As for your buffer extraction, I thought it odd about those intermediate returned Buffers, will be more mindful about those functions. I suppose I should work on a function that extracts from a buffer without any leak:

string fExtract(Buffer buf, int iFrom, iTo)
{ return(string bufiFrom, iTo) //??
}

bufResults += NameMod
bufResults += "\t"
bufResults += fExtract(bufInfo, 3, 22)
bufResults += "\n"

These do not produce leaks do they?
Buffer ::+=(Buffer,char) BufferCatChar
Buffer ::+=(Buffer,Buffer) BufferCatBuf
Buffer ::+=(Buffer,string) BufferCat

  • Louie

Ok, the string table issue is easy to reproduce
  

// will create a buffer with length 2^26 = 67108864 = 65 MB
Buffer b = create(68000000); 
 
// change this character!
// the 65 MB string will reside in the string table, but only if it does
// not already exist
b += 'a' 
 
for i in 0:25 do combine(b, b, 0) 
string s = stringOf b 
delete b 
ack (length s) " leaked ... Look at the virtual memory consumption in task manager"

That is the most probably source of memory leaks that come from string handling with memory staying alive even after the DXL closes. You should always use combine to add a substring of a buffer to another buffer, by the way.

The GUI thing happens when people do another common error, destroying the GUI from a callback: 

DB x = create (current Module, "Memory Leak" )
 
Buffer b = create(68000000)
b += 'a'
for i in 0:25 do combine(b, b, 0) 
 
void doCB(DBE xx) { 
   // destroy from callback will leak the memory. Use hide instead!
   DB x = getParent xx; destroy x 
}
 
button(x, "Leak", doCB) 
 
show x

The += operators do not produce intermediate buffers (so they do not leak memory). To find memory leaks by script you can use the below hack: 

int *::+(int *ptr1, int ofs) { int *ptr2 = ptr1; ptr2+=ofs; return ptr2 }
int *::@(int *ptr, int ofs) { int ad = *(ptr + ofs); int *ptr2 = addr_ ad; return ptr2 }
 
int *ccp () {
    DB x = create ""
        int *p = addr_ x; int *r = p @ 48
        destroy x
        return r
}
 
int *mbn (int *cc) { return cc @ 0x74 }
int *dp (int *n) { return n @ 0 }
int *nn      (int *n) { return n @ 8 }
 
monitorWidth = (addr_ ccp()) int
userNameWidth = -1
 
void debCB(string s, int ln) {
   int cnt = 0
   if (ln == 0) return
   int *dc = addr_ monitorWidth
   int *mb = mbn dc 
   while (!null mb) { mb = nn mb; cnt++ }
   if (cnt != userNameWidth) {
     userNameWidth = cnt
     print s ":" ln " --> " cnt "\n"
   }
 
}
 
setDebugCallback_ (debCB)
setDebugging_ (true)
 
// ******************** our program here **********************
 
int i
 
Buffer b = create(); b = "1234567890"
string s
 
for i in 0:10 do {
        s = "Hallo " b[3:6] ""
        print s "\n"
}
 
// ************************************************************
 
setDebugging_ (off)


As always ignore the weird lines, just insert the program to debug between the comments. It will run much slower than before, since the debugging callback is tracking the allocated objects in the current DXL context and will print you the line of code, where the number of allocated objects is increased. This will give you an output like: 

Line:38 --> 0
Line:38 --> 1
Line:42 --> 2
Hallo 4567
Line:42 --> 3
Hallo 4567
Line:42 --> 4
Hallo 4567
Line:42 --> 5
Hallo 4567
Line:42 --> 6
Hallo 4567
Line:42 --> 7
Hallo 4567


where 42 is the line where the buffer is leaked and 38 the first line of execution (first print) and again where the buffer is created.

I hope this lengthy answer will help :-) Regards, Mathias

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

 

 

Re: Massive Memory Leak
David_G_Bond - Mon Jun 20 16:10:57 EDT 2011

Mathias Mamsch - Mon Jun 20 15:47:34 EDT 2011

Ok, the string table issue is easy to reproduce
  

// will create a buffer with length 2^26 = 67108864 = 65 MB
Buffer b = create(68000000); 
 
// change this character!
// the 65 MB string will reside in the string table, but only if it does
// not already exist
b += 'a' 
 
for i in 0:25 do combine(b, b, 0) 
string s = stringOf b 
delete b 
ack (length s) " leaked ... Look at the virtual memory consumption in task manager"

That is the most probably source of memory leaks that come from string handling with memory staying alive even after the DXL closes. You should always use combine to add a substring of a buffer to another buffer, by the way.

The GUI thing happens when people do another common error, destroying the GUI from a callback: 

DB x = create (current Module, "Memory Leak" )
 
Buffer b = create(68000000)
b += 'a'
for i in 0:25 do combine(b, b, 0) 
 
void doCB(DBE xx) { 
   // destroy from callback will leak the memory. Use hide instead!
   DB x = getParent xx; destroy x 
}
 
button(x, "Leak", doCB) 
 
show x

The += operators do not produce intermediate buffers (so they do not leak memory). To find memory leaks by script you can use the below hack: 

int *::+(int *ptr1, int ofs) { int *ptr2 = ptr1; ptr2+=ofs; return ptr2 }
int *::@(int *ptr, int ofs) { int ad = *(ptr + ofs); int *ptr2 = addr_ ad; return ptr2 }
 
int *ccp () {
    DB x = create ""
        int *p = addr_ x; int *r = p @ 48
        destroy x
        return r
}
 
int *mbn (int *cc) { return cc @ 0x74 }
int *dp (int *n) { return n @ 0 }
int *nn      (int *n) { return n @ 8 }
 
monitorWidth = (addr_ ccp()) int
userNameWidth = -1
 
void debCB(string s, int ln) {
   int cnt = 0
   if (ln == 0) return
   int *dc = addr_ monitorWidth
   int *mb = mbn dc 
   while (!null mb) { mb = nn mb; cnt++ }
   if (cnt != userNameWidth) {
     userNameWidth = cnt
     print s ":" ln " --> " cnt "\n"
   }
 
}
 
setDebugCallback_ (debCB)
setDebugging_ (true)
 
// ******************** our program here **********************
 
int i
 
Buffer b = create(); b = "1234567890"
string s
 
for i in 0:10 do {
        s = "Hallo " b[3:6] ""
        print s "\n"
}
 
// ************************************************************
 
setDebugging_ (off)


As always ignore the weird lines, just insert the program to debug between the comments. It will run much slower than before, since the debugging callback is tracking the allocated objects in the current DXL context and will print you the line of code, where the number of allocated objects is increased. This will give you an output like: 

Line:38 --> 0
Line:38 --> 1
Line:42 --> 2
Hallo 4567
Line:42 --> 3
Hallo 4567
Line:42 --> 4
Hallo 4567
Line:42 --> 5
Hallo 4567
Line:42 --> 6
Hallo 4567
Line:42 --> 7
Hallo 4567


where 42 is the line where the buffer is leaked and 38 the first line of execution (first print) and again where the buffer is created.

I hope this lengthy answer will help :-) Regards, Mathias

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

 

 

Are you using DOORS 9.x? We are having similar problems (see my earlier post). I have also learned that the spell checker exhausts memory when used in large modules. I wonder if these are all related problems.

  • David Bond

Re: Massive Memory Leak
llandale - Mon Jun 20 22:35:01 EDT 2011

David_G_Bond - Mon Jun 20 16:10:57 EDT 2011
Are you using DOORS 9.x? We are having similar problems (see my earlier post). I have also learned that the spell checker exhausts memory when used in large modules. I wonder if these are all related problems.

  • David Bond

Partners using v8.3. I tried to duplicate their situation (27k+ outlinks) and can see the increased in memory loss in v9.2. Tried to reduce number of open modules but that wasn't it.

Re: Massive Memory Leak
SystemAdmin - Thu Nov 10 09:35:55 EST 2011

Mathias Mamsch - Mon Jun 20 15:47:34 EDT 2011

Ok, the string table issue is easy to reproduce
  

// will create a buffer with length 2^26 = 67108864 = 65 MB
Buffer b = create(68000000); 
 
// change this character!
// the 65 MB string will reside in the string table, but only if it does
// not already exist
b += 'a' 
 
for i in 0:25 do combine(b, b, 0) 
string s = stringOf b 
delete b 
ack (length s) " leaked ... Look at the virtual memory consumption in task manager"

That is the most probably source of memory leaks that come from string handling with memory staying alive even after the DXL closes. You should always use combine to add a substring of a buffer to another buffer, by the way.

The GUI thing happens when people do another common error, destroying the GUI from a callback: 

DB x = create (current Module, "Memory Leak" )
 
Buffer b = create(68000000)
b += 'a'
for i in 0:25 do combine(b, b, 0) 
 
void doCB(DBE xx) { 
   // destroy from callback will leak the memory. Use hide instead!
   DB x = getParent xx; destroy x 
}
 
button(x, "Leak", doCB) 
 
show x

The += operators do not produce intermediate buffers (so they do not leak memory). To find memory leaks by script you can use the below hack: 

int *::+(int *ptr1, int ofs) { int *ptr2 = ptr1; ptr2+=ofs; return ptr2 }
int *::@(int *ptr, int ofs) { int ad = *(ptr + ofs); int *ptr2 = addr_ ad; return ptr2 }
 
int *ccp () {
    DB x = create ""
        int *p = addr_ x; int *r = p @ 48
        destroy x
        return r
}
 
int *mbn (int *cc) { return cc @ 0x74 }
int *dp (int *n) { return n @ 0 }
int *nn      (int *n) { return n @ 8 }
 
monitorWidth = (addr_ ccp()) int
userNameWidth = -1
 
void debCB(string s, int ln) {
   int cnt = 0
   if (ln == 0) return
   int *dc = addr_ monitorWidth
   int *mb = mbn dc 
   while (!null mb) { mb = nn mb; cnt++ }
   if (cnt != userNameWidth) {
     userNameWidth = cnt
     print s ":" ln " --> " cnt "\n"
   }
 
}
 
setDebugCallback_ (debCB)
setDebugging_ (true)
 
// ******************** our program here **********************
 
int i
 
Buffer b = create(); b = "1234567890"
string s
 
for i in 0:10 do {
        s = "Hallo " b[3:6] ""
        print s "\n"
}
 
// ************************************************************
 
setDebugging_ (off)


As always ignore the weird lines, just insert the program to debug between the comments. It will run much slower than before, since the debugging callback is tracking the allocated objects in the current DXL context and will print you the line of code, where the number of allocated objects is increased. This will give you an output like: 

Line:38 --> 0
Line:38 --> 1
Line:42 --> 2
Hallo 4567
Line:42 --> 3
Hallo 4567
Line:42 --> 4
Hallo 4567
Line:42 --> 5
Hallo 4567
Line:42 --> 6
Hallo 4567
Line:42 --> 7
Hallo 4567


where 42 is the line where the buffer is leaked and 38 the first line of execution (first print) and again where the buffer is created.

I hope this lengthy answer will help :-) Regards, Mathias

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

 

 

Hi Mathias,

I was playing with your code for walking the Allocated Objects List, and wondered why you are using 'monitorWidth' and 'userNameWidth'.

I tried changing 'monitorWidth' to an int variable instead but that crashes DOORS.

I also tried getting the DXLContextPtr(*cpp) in the Debug Callback rather than using a global but that causes the Count to increase alot.

Could you explain why that happens?

Re: Massive Memory Leak
Mathias Mamsch - Fri Nov 11 04:15:12 EST 2011

SystemAdmin - Thu Nov 10 09:35:55 EST 2011
Hi Mathias,

I was playing with your code for walking the Allocated Objects List, and wondered why you are using 'monitorWidth' and 'userNameWidth'.

I tried changing 'monitorWidth' to an int variable instead but that crashes DOORS.

I also tried getting the DXLContextPtr(*cpp) in the Debug Callback rather than using a global but that causes the Count to increase alot.

Could you explain why that happens?

Oh, that is a complex one ;-) The idea was to register a debug callback that would be able to count the number of allocated objects after each DXL statement, to see where the increases happen. This is an alternative approach to hooking all allocation functions and explicitly count allocations. I found the hooking approach to work very reliable, but I did not investigate in the debug callback approach to much yet. Anyway, when implementing this I noticed that I could not access my program variables from the Debug callback and I got crashes. That hinted me to the conclusion that the Debug Callback is evaluated from either another or the global DXL context. The following code shows the problem:
  

int *::+(int *ptr1, int ofs) { int *ptr2 = ptr1; ptr2+=ofs; return ptr2 }
int *::@(int *ptr, int ofs) { int ad = *(ptr + ofs); int *ptr2 = addr_ ad; return ptr2 }
 
int *currentDXLContext() {
    DB x = create ""
        int *p = addr_ x; int *r = p @ 48
        destroy x
        return r
}
 
print "From the program context:" ((addr_ currentDXLContext()) int) "\n"
 
void debCB(string s, int ln) { print "From the debug context:" ((addr_ currentDXLContext()) int) "\n" }
 
setDebugCallback_ (debCB)
setDebugging_ (true)
setDebugging_ (false)

 


Therefore the question was how I could get the list of allocated objects from the program context, from the debug callback, since inside the debug callback I was in another context. The solution was to use a variable of the top-level context, but I wanted a solution that would not interfere with the DOORS client, by changing some of its variables to some random value.

Therefore I looked through the standard library code to find a global variable, that is not used. I found it with MonitorWidth and UserNameWidth which were only used to build the Database explorer GUI, but not anywhere else. So they seemed free for misusing. ;-)

I put the DXLContext pointer from the main program into that global variable and use that one instead of getting the current DXL context pointer. Additionally I misused the other one for the counter, but you can always put a Skip list in there to store more stuff. You need to be careful however, anything that you allocate in the debug callback will be deallocated after the call (just like with eval). Therefore you should not for example construct strings inside the debug callback and store them to a global reference. Instead copy them to a global buffer.

By the way, if you really want to use that code to find memory leaks you probably need to spice it up a little. You should count the number of allocations for each line. Addionally you need to store the last line number, because obviously the allocation happened one line before the current line. If you just returned from a function or stepped inside a function (like "func (create Buffer())") this might be misleading.

Hope that clears things up, regards, Mathias

 

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

 

Re: Massive Memory Leak
SystemAdmin - Fri Nov 11 05:00:19 EST 2011

Mathias Mamsch - Fri Nov 11 04:15:12 EST 2011

Oh, that is a complex one ;-) The idea was to register a debug callback that would be able to count the number of allocated objects after each DXL statement, to see where the increases happen. This is an alternative approach to hooking all allocation functions and explicitly count allocations. I found the hooking approach to work very reliable, but I did not investigate in the debug callback approach to much yet. Anyway, when implementing this I noticed that I could not access my program variables from the Debug callback and I got crashes. That hinted me to the conclusion that the Debug Callback is evaluated from either another or the global DXL context. The following code shows the problem:
  

int *::+(int *ptr1, int ofs) { int *ptr2 = ptr1; ptr2+=ofs; return ptr2 }
int *::@(int *ptr, int ofs) { int ad = *(ptr + ofs); int *ptr2 = addr_ ad; return ptr2 }
 
int *currentDXLContext() {
    DB x = create ""
        int *p = addr_ x; int *r = p @ 48
        destroy x
        return r
}
 
print "From the program context:" ((addr_ currentDXLContext()) int) "\n"
 
void debCB(string s, int ln) { print "From the debug context:" ((addr_ currentDXLContext()) int) "\n" }
 
setDebugCallback_ (debCB)
setDebugging_ (true)
setDebugging_ (false)

 


Therefore the question was how I could get the list of allocated objects from the program context, from the debug callback, since inside the debug callback I was in another context. The solution was to use a variable of the top-level context, but I wanted a solution that would not interfere with the DOORS client, by changing some of its variables to some random value.

Therefore I looked through the standard library code to find a global variable, that is not used. I found it with MonitorWidth and UserNameWidth which were only used to build the Database explorer GUI, but not anywhere else. So they seemed free for misusing. ;-)

I put the DXLContext pointer from the main program into that global variable and use that one instead of getting the current DXL context pointer. Additionally I misused the other one for the counter, but you can always put a Skip list in there to store more stuff. You need to be careful however, anything that you allocate in the debug callback will be deallocated after the call (just like with eval). Therefore you should not for example construct strings inside the debug callback and store them to a global reference. Instead copy them to a global buffer.

By the way, if you really want to use that code to find memory leaks you probably need to spice it up a little. You should count the number of allocations for each line. Addionally you need to store the last line number, because obviously the allocation happened one line before the current line. If you just returned from a function or stepped inside a function (like "func (create Buffer())") this might be misleading.

Hope that clears things up, regards, Mathias

 

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

 

I didn't even consider that the DebugCallback might be executing in a different context. I first thought the creation of a local variable before getting the ContextPtr may be causeing the memory offset from the DB to change - but as the DB is on the heap, that makes no sense. Then I thought the memory offset to the LinkedList from the Context may be changed - but getting it all in the Callback works, but gives the wrong number of AllocatedObjects. From your explanation, that was obviously the number of AllocatedObjects in the GlobalContext.

Thanks for the explaination - makes sense now.

Re: Massive Memory Leak
SystemAdmin - Mon Nov 21 08:29:38 EST 2011

Mathias Mamsch - Fri Nov 11 04:15:12 EST 2011

Oh, that is a complex one ;-) The idea was to register a debug callback that would be able to count the number of allocated objects after each DXL statement, to see where the increases happen. This is an alternative approach to hooking all allocation functions and explicitly count allocations. I found the hooking approach to work very reliable, but I did not investigate in the debug callback approach to much yet. Anyway, when implementing this I noticed that I could not access my program variables from the Debug callback and I got crashes. That hinted me to the conclusion that the Debug Callback is evaluated from either another or the global DXL context. The following code shows the problem:
  

int *::+(int *ptr1, int ofs) { int *ptr2 = ptr1; ptr2+=ofs; return ptr2 }
int *::@(int *ptr, int ofs) { int ad = *(ptr + ofs); int *ptr2 = addr_ ad; return ptr2 }
 
int *currentDXLContext() {
    DB x = create ""
        int *p = addr_ x; int *r = p @ 48
        destroy x
        return r
}
 
print "From the program context:" ((addr_ currentDXLContext()) int) "\n"
 
void debCB(string s, int ln) { print "From the debug context:" ((addr_ currentDXLContext()) int) "\n" }
 
setDebugCallback_ (debCB)
setDebugging_ (true)
setDebugging_ (false)

 


Therefore the question was how I could get the list of allocated objects from the program context, from the debug callback, since inside the debug callback I was in another context. The solution was to use a variable of the top-level context, but I wanted a solution that would not interfere with the DOORS client, by changing some of its variables to some random value.

Therefore I looked through the standard library code to find a global variable, that is not used. I found it with MonitorWidth and UserNameWidth which were only used to build the Database explorer GUI, but not anywhere else. So they seemed free for misusing. ;-)

I put the DXLContext pointer from the main program into that global variable and use that one instead of getting the current DXL context pointer. Additionally I misused the other one for the counter, but you can always put a Skip list in there to store more stuff. You need to be careful however, anything that you allocate in the debug callback will be deallocated after the call (just like with eval). Therefore you should not for example construct strings inside the debug callback and store them to a global reference. Instead copy them to a global buffer.

By the way, if you really want to use that code to find memory leaks you probably need to spice it up a little. You should count the number of allocations for each line. Addionally you need to store the last line number, because obviously the allocation happened one line before the current line. If you just returned from a function or stepped inside a function (like "func (create Buffer())") this might be misleading.

Hope that clears things up, regards, Mathias

 

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

 

Been playing with this a bit more.

Just wondering why we are able to call functions defined in our context(eg your mbn function) from the DebugCallback, when the variables defined globally in the same context can't be used.

Any ideas on that?

Re: Massive Memory Leak
Mathias Mamsch - Thu Nov 24 07:12:23 EST 2011

SystemAdmin - Mon Nov 21 08:29:38 EST 2011
Been playing with this a bit more.

Just wondering why we are able to call functions defined in our context(eg your mbn function) from the DebugCallback, when the variables defined globally in the same context can't be used.

Any ideas on that?

The DXL interpreter converts the DXL program into an internal data structure (sometimes called 'XTC'), that it will process using some kind of 'virtual machine' (just like byte code with java). You can see a representation of these micro operations using the progPrint_ perm. Therefore you can see that as long as the bytecode for a function is in memory it is no problem to 'call' it, i.e. to execute it for the DXL interpreter regardless of to which context the program belongs.

The important thing is how the DXL Execution Engine handles the stack, literals that are embedded inside a program and variables that are declared inside a program. Literals and constants will be placed in a separate memory region at compile time. That is during interpretation the DXL Parser will put a literal string embedded in your program somewhere in the memory and then replace the reference to that string by that memory location. Every variable you declare gets some part on the stack. Additionally DXL keeps a symbol table, where it stores the names to those variables during compile time. Then every reference to that variable will be replaced by a stack location at compile time. That means you can call your function that was compiled in one context in another context.
  

int global = 99
void funcMain(int x) { print "Calling Main:" x " Global:" global "\n" }
string codeChild = "void proxyChild(void func(int), int a) { func a } 
int ad = (addr_ " ((addr_ funcMain) int) ") int 
proxyChild (addr_ ad, 15)"
 
print "Evaluating:\n" codeChild "\n-----------------------\n"
eval_ codeChild

 


When you call the function, the stack of the executing context will be used, that is NOT the context in which the function resides. That is no problem for parameters you pass, since the call and therefore the putting the parameters to the stack is in the executing context, so the relative address (stack pointer - 8), (stack pointer - 4), etc. are still valid parameters (in the example the parameter 15). Note that it is declared in the eval context. The 'global' variable however is defined in a different context. From the perspective of the main context, this is the first variable declared, so the reference in the function got replaced by "top of the stack".
In the above example of eval_ context and main context this is not a problem, since the namespaces are separate and I cannot access the global variables of the main program from the eval_ context. But in case of the debug function the funny thing is, that the code inside the function will be called from a different DXL context but resides in the same code, that is all stack locations will not be valid anymore. I am not sure if the top level context is used to execute the debug function or a separate context.

Maybe that helps, regards, Mathias

 

 

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS